Method and apparatus for displaying real-time visual information on an automobile pervasive computing client

ABSTRACT

A navigation system, preferably for use in a vehicle. The system includes a processor, and a graphical display for displaying map information. As the vehicle approaches a given location, e.g., an intersection, a visual image of the location is retrieved and displayed on the graphical display. In one embodiment, the visual image is a photograph of the location that is displayed in a pop-up window on the display screen. If desired, additional graphic images or text are superimposed on or associated with the image to facilitate navigation. Thus, for example, if the user were approaching an intersection, the invention displays a still photograph of the intersection, together with an arrow overlaid thereupon to illustrate that the user should make a given turn.

BACKGROUND OF THE INVENTION

1. Technical Field

This invention relates generally to in-vehicle navigation systems and,in particular, to a method and system for displaying visual images on aportable computing device to assist a user in locating trafficlandmarks.

2. Description of the Related Art

Recently, the computer industry has sought to add computer processingand communications capabilities to devices other than what wouldnormally be considered a traditional computer. Such devices are quitevaried and include, for example, personal digital assistants (PDAs),smartphones, cellular phones, desktop screen phones, in-vehicle devices,vehicle traffic lights, kiosks, business organizers (e.g., IBMWorkPadT™, PalmPilot™, and the like), computer peripherals (such asprinters, fax machines, and the like), handheld or palmtop computingdevices, and the like. For convenience, these devices, as a class, arereferred to herein as “pervasive computing” clients as they are devicesthat are designed to be connected to servers in a computer network andused for computing purposes regardless of their location.

Palmtop computers and the like are now being proposed as pervasivecomputing devices for use in an automobile. In-vehicle navigationsystems, of course, are well-known. Representative systems are describedin U.S. Pat. Nos.: 5,121,326, 5,191,532 and 5,792,109. U.S. Pat. No.5,121,326 describes a navigation system that displays a general map ofthe driver's location. As the driver approaches an intersection, thedisplay provides a detailed map of the intersection to provide furtherguidance to the driver. In U.S. Pat. No. 5,191,532, a navigation systemis disclosed where driving directions can be provided even though theuser's exact destination is not listed in the systems database. Certaingeographic features such as cities and landmarks are used to approximatethe destination for purposes of providing directions. In U.S. Pat. No.5,729,109, the inventors provide a vehicle navigation system that usesspeech generation to give spoken directions when the driver isapproaching an intersection.

Although such existing in-vehicle navigation systems are useful fortheir intended purpose, the user interface for such devices is oftenfairly crude. Typically, the interface comprises a grid or mapillustrating various roads, streets and intersections. Occasionally,these maps include other identifying information. Thus, while a user maybe provided with some useful information from such displays, maps areoften confusing, especially at some complex intersections. Moreover,these types of displays generally do not always provide sufficientinformation so as to enable a user to accurately determine if thevehicle has reached a given location represented on the map.

The present invention addresses the need to provide improved in-vehicledisplay of navigation information.

BRIEF SUMMARY OF THE INVENTION

An object of the present invention is to provide accurate visualinformation to a driver in using a navigation system to allow the driverto more accurately assess his or her position relative to anintersection or other physical location where an action may need to betaken.

It is a further object to provide a vehicle navigation system with thecapability of displaying a visual identification (e.g., a photograph)consistent with the driver's then-current perspective to allow for moreaccurate and intuitive decision making during the course of navigating avehicle.

It is a yet another object of the present invention to provide a moreuseful in-vehicle navigation system that displays informationautomatically so that a user has more complete data available at anearliest possible time prior to a decision point.

A more general object of the present invention is to provide an improvedvehicle navigation display system and interface.

Another general object of this invention is to provide still photographsor real-time videos of physical landmarks in conjunction with display ofa navigation grid or map in a vehicle or handheld navigation system.

These and other objects of the invention are provided in a navigationsystem, preferably for use in a vehicle. The system includes aprocessor, and a graphical display for displaying map information. Asthe vehicle approaches a given location, e.g., an intersection, a visualimage of the location is retrieved and displayed on the graphicaldisplay. In one embodiment, the visual image is a photograph of thelocation that is displayed in a pop-up window on the display screen. Ifdesired, additional graphic images or text are superimposed on orassociated with the image to facilitate navigation. Thus, for example,if the user were approaching an intersection, the invention displays astill photograph of the intersection, perhaps with an arrow overlaidthereupon to illustrate that the user should make a given turn.

Each location may have a set of photograph images associated therewith.Thus, for example, a given intersection is photographed at differenttimes of day and/or at different times of the year. This enables theuser to be provided with the most accurate representation of thelocation as the user approaches. If sufficient processing capability isavailable in the vehicle navigation system and the content is otherwiseavailable, the system may be selectively controlled to display videoimages of the physical location. This would be quite helpful in thesituation wherein physical elements in the scenery are consistentlychanging.

Preferably, the image content is collected by other vehicles (e.g.,police cars, delivery vehicles, taxi cars, post office vehicles, etc.)and stored on given physical media within a user's vehicle (e.g., as aCD-ROM, or the like), or such information may be stored in a server. Inthe latter case, the in-vehicle system connects to the server (e.g., viaa wireless connection) and receives the desired content for display onthe user's computer.

The inventive display method may also be practiced in otherenvironments. Thus, for example, the method may be implemented in ahandheld or mobile computer that includes a global positioning system(GPS) or the like for displaying the position of the device. In suchcase, a grid or map may be displayed on the computer screen. When theuser approaches some location of interest, a photographic image isdisplayed, e.g., in a pop-up window, as a visual association of theuser's position with some actual physical location.

The foregoing has outlined some of the more pertinent objects andfeatures of the present invention. These objects should be construed tobe merely illustrative of some of the more prominent features andapplications of the invention. Many other beneficial results can beattained by applying the disclosed invention in a different manner ormodifying the invention as will be described. Accordingly, other objectsand a fuller understanding of the invention may be had by referring tothe following Detailed Description of the Preferred Embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention and theadvantages thereof, reference should be made to the following DetailedDescription taken in connection with the accompanying drawings in which:

FIG. 1 is a view of a example pervasive computing device useful forimplementing the present invention;

FIG. 2 is a diagram of a representative client-server architecture inwhich the pervasive computing client of the present invention is used;

FIG. 3 is a schematic diagram of the pervasive computing device asconnected in a vehicle;

FIG. 4 is an illustrative display of a map provided in a representativeembodiment of the present invention;

FIG. 5 is the illustrative display of FIG. 4 that has been augmented todisplay a photographic image according to the teachings of the presentinvention; and

FIG. 6 is a flowchart of a preferred control routine of the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention provides a novel navigation data processing anddisplay device, system and method. Preferably, the invention isimplemented in a mobile computer, such as a pervasive computing client,connectable to a computer network. FIG. 1 illustrates a representativepervasive computing client device 10 having a graphical display 12. Thedevice also includes a handheld stylus 15 for inputting information tothe device. A representative handheld device in which the presentinvention is implemented is a palmtop computer, for example, a devicemarketed by the IBM Corporation under the WorkPad trademark. One ofordinary skill, however, will appreciate that the principles of theinvention are generally applicable to a pervasive computing client.Representative devices include a pervasive client that is x86-,PowerPC®- or RISC-based, that includes a real-time operating system suchas WindRiver VXWorks™, QSSL QNXNeutrino™, or Microsoft Windows CE, andthat includes a browser.

Referring now to FIG. 2, a representative pervasive computing devicecomprises client stack 20 including a number of components, for example,a client application framework 22, a virtual machine 24, a speech engine26, and an industry-supplied runtime operating system (RTOS) 28. Theclient application framework 22 typically includes a browser 30, a userinterface 32, a pervasive computing client application class library 34,a standard Java class library 36, and a communication stack 38. Thepervasive computing client connects to a server platform 40 via awireless data link 42.

The pervasive computing client preferably includes a global positioningsystem (GPS) receiver 45. Recently, such receivers have become readilyavailable on a widespread commercial basis as integrated circuitdevices. Such devices provide latitude and longitude data to within agiven tenth of a second of arc. When greater accuracy is required, theaccuracy of the position measurement is enhanced using differential GPS(or dGPS), a process in which a number of fixed reference points areused. In dGPS, the positions of the reference points are determined withgreat precision, e.g., using surveying techniques. A GPS is used toobtain the location of a given reference point, and this measurement iscompared with known location to generate a correction value that is thenused to correct the position of the pervasive device as measured by GPS.

Returning back to FIG. 2, at its lower level, the connectivity service31 includes a gateway 33 that provides compression and encryptionfunctions. The upper level of the connectivity service 31 is a proxy 35that provides several different functions: transcoding, filtering,prioritization and link to device management. Transcoding refers to thetranslation from one source markup language to another markup language.Transcoding is required because a pervasive computing client normallydoes not support the full function set of an HTML Windows-based client.In such case, it is necessary to transcode the HTML-based file into aformat (e.g., HDML or handheld device markup language) compatible withthe pervasive client computing device so that the file may beappropriately rendered on the client.

The server platform 40 may be of several different types. The platform40 may be a Web/application server 37 (a synchronous request-responsetype server) or a data synchronization server 39 (an asynchronous queuedcommunication type server). The basic functions are each such servertype are illustrated. Alternatively, the platform 40 may be avalue-added server 41 that provides additional services such as LDAPdirectory/repository, awareness and notification, network management,device life cycle management, user and device registration, or billing.

The particular server is typically one of a plurality of servers whichare accessible by clients, one of which is illustrated by the pervasivecomputing client having a browser, as previously noted. A representativeWeb server is an IBM Netfinity™ server comprising a RISC-basedprocessor, the AIX® operating system and a Web server program, such asNetscape Enterprise Server.

A representative server also includes a display supporting a graphicaluser interface (GUI) for management and administration, and anApplication Programming Interface (API) that provides extensions toenable application developers to extend and/or customize the corefunctionality thereof through software programs including Common GatewayInterface (CGI) programs, plugins, servlets, active server pages, serverside include (SSI) functions or the like.

The inventive functionality of the pervasive computing client 10 ispreferably implemented in a software executable in a processor, namely,as a set of instructions (program code) in a code module resident in therandom access memory of the computer. Until required by the computer,the set of instructions may be stored in another computer memory, forexample, in a hard disk drive, or in a removable memory, or downloadedvia the Internet or other computer network. This program is preferablyinvoked by pressing an icon 14 using the stylus 15.

According to a preferred embodiment, the pervasive computing client is aportable or so-called “palmtop” computer or PDA. Generally, the portablecomputing device of the invention is used as an in-vehicle navigationcomputer to display to the driver (or occupant) maps and other usefulinformation. According to the invention, such other informationpreferably includes rich media content, such as a photographic image ofa landmark located at an upcoming intersection or other location. Suchcontent may be downloaded to the vehicle computer from a server in thenetwork (as previously illustrated) or, alternatively, such content maybe pre-recorded on a storage medium (e.g., CD-ROM, DVD, or the like)provided in the vehicle.

The in-vehicle embodiment of the present invention is now described inmore detail. In particular, FIG. 3 is a block diagram of an navigationsystem wherein a pervasive computing device 10 is mounted on a cradle 51which serves at least two purposes. First, the cradle is positioned inthe vehicle (not shown) to allow for good visibility of graphicaldisplay 12 to the operator or navigator of the vehicle. Second, cradle51 provides electrical connection from the pervasive computing device toother devices mounted in the vehicle. For example, vehicles are usuallyequipped with high capacity power storage devices. Battery 52 isconnected (through an appropriate regulator, not shown) to pervasivecomputing device 10. Another device connected to the pervasive devicevia cradle 51 is mass storage device 54. In this example, mass storagedevice 54 is preferably a DVD type optical disk drive. DVD disks, whenused for data storage, provide very large storage capacity. Thiscapacity is valuable in storing the high quality images that arepreferably used in implementing the present invention. Mass storagedevice may be other types of storage devices such as a CD-ROM, a highcapacity hard disk drive or flash memory card.

Cradle 51 also allows for connection to the GPS module 45. The use ofthe in-vehicle global positioning system (GPS) is well-known in the art,as illustrated in U.S. Pat. No. 5,862,511.

According to an alternate embodiment, the inventive display routine isimplemented within a handheld pervasive computing device. Such a devicemay include a GPS receiver or, alternatively, internal positioningdevices (e.g., an accelerometer, a pedometer, a compass, and the like).Such devices may be used to provide position and direction informationto the device. In addition, the device typically includes mappingroutines to illustrate various characteristics of a given geographicarea. Such a mapping routine, for example, would generate an overlayimage illustrating physical features and characteristics, contours,property lines, dwellings, roads, waterways, and the like. Arepresentative system of this type is described in U.S. Pat. No.5,699,244.

Cradle 51 also provides a connection between pervasive computing device10 and wireless data transceiver 58. Wireless data transceiver 58 allowsfor establishment of wireless data link (as seen in FIG. 2) to a datanetwork allowing establishment of client/server communications to andfrom server 40. The communications link can be provided using one ofseveral known wireless data transmission technologies such as CDMA, GSM,TDMA, CDPD, and Mobitex.

FIG. 4 depicts an illustrative display on graphical display 12 of thepervasive device. In this example, direction arrow 80 indicates thedirection of travel along a road 82. The destination is represented, forexample, by cross 84. Other features such as roads 86 and lake 88 arealso depicted. Preferably, the vehicle's location along the route isdisplayed and updated continuously in real-time or according to somegiven time interval (e.g., every 5 seconds). For the sake of thisexample, an important location on the vehicle's desired route isintersection 90. In developing the maps displayed using this invention,certain points may be designated as difficult to navigate. Intersection90 is one of those points in this example.

When the vehicle reaches a selected distance (or time) away fromintersection 90, a display function in the device is invoked to fetch aphotographic image 92 of intersection 90 depicting the impending scene.This image is then displayed as shown in FIG. 5. The graphical image 92in this example provides aid to the driver of the vehicle by allowinghim or her to see landmarks, such as water tower 94 and tree 96, at thecritical location 90 in their natural perspective. The image may be ofany convenient format (e.g., .jpeg, gif, .png. or the like) that may bereadily transmitted (if required), stored and displayed. In anillustrative embodiment, the image is displayed in a pop-up window 93 ongraphical display 12. Preferably, the window is generated by thedevice's browser. Alternatively, if the device does not include abrowser, a separate window may be created by the device's operatingsystem.

In accordance with the present invention, the photographic image ispreferably as close as possible to the real-time conditions that thedriver is encountering during the drive. Thus, for example, if thedriver is approaching the location at night, or during the Wintermonths, the photograph displayed represents the most accuraterepresentation of the physical area. In this way, the device providesthe user with both a graphical representation or navigation map,together with an actual representation of what physical landmarks arecoming into the user's field of vision. If desired, the graphicaldisplay may provide other reference aids, such as display graphics (suchas arrows and pointers) or the like, to further assist the user tonavigate to the desired location. Alternatively, the display system maygenerate a voice-over (e.g., using a text-to-speech processor) thatgenerates an audio cue.

To provide a concrete example, the in-vehicle display system displays agrid, a photograph, and a visual cue. An audio overlay (e.g., “as youcan see by the image, you are now arriving at the intersection of Mainand Commerce. You should now turn right”). As illustrated in FIG. 5,direction arrow 96 is superimposed on the graphical image to further aidto the driver.

Given images to be displayed are retrieved from data storage in thevehicle (e.g., in a CD-ROM or DVD format) or are otherwise selectivelydownloaded to the pervasive computing client via the wireless network.

Preferably, the given photographic image is displayed when the userreaches a given position relative to the location of interest. Thus, inthe example of FIGS. 4-5, the photographic image is displayed at acertain point in time, or when the vehicle reaches a certain position inadvance of the intersection, to enable the driver to have sufficienttime (given the driving conditions, the speed or travel and thedistance) to make an informed navigation decision.

These photographs can be gathered for the navigational system in anumber of ways. A preferable way is the use of a digital camera (such asthat shown in U.S. Pat. No. 5,859,666, which is incorporated herein byreference, mounted to vehicles which regularly travel the area, forexample, police, public transportation, emergency, taxi or deliveryvehicles. The use of such cameras with police vehicles is particularlyadvantageous because photographic evidence taken from police vehiclesaids the police in their work.

In an illustrative embodiment, the photographs are collected as follows.Additional details regarding this technique in Applicant's copendingapplication titled “Video Camera With Position And Orientation RecordingTo Assist In Authoring Virtual Reality Scenes.” That application isincorporated herein by reference. It is identified as Ser. No.09/240,925. In that technique, a video camera is provided on a trackingvehicle. The video camera is augmented with a set of accelerometers or,alternatively, accelerometers in combination with a set of gyroscopes,and, optionally, a GPS receiver. For each frame of video picture taken,the camera's location and orientation are recorded along with thepicture. The video is subsequently digitized, possibly compressed, andstored in a computer's hard disk drive or other storage media. Theframes are then sorted out by location and orientation and organizedinto an accessible data structure. This information is then subsequentlyused to facilitate display of the additional navigation images as hasbeen previously described.

In a preferred embodiment, graphical image 92 is stored in a local massstorage device such as mass storage device 54. Preferably, mass storagedevice 54 is a DVD ROM device. Disks can be stored which provide theimages for an entire region. Local storage of the images provides forrapid retrieval and display. An alternative is to store the images in aserver system provided by server 40 (as in FIG. 3). This approachprovides nearly limitless storage, but it may have lower image accessrates depending on the transfer rate of the wireless data link.

Alternatively, the graphical image 92 may be a moving video imageillustrating landmarks located at the intersection. As the driverapproaches, the video will change the perspective to provide a moreaccurate viewpoint. This provides more accurate data to the driver, butrequires more data. Thus, this alternative would increase the cost andcomplexity of the system.

FIG. 6 is a flowchart depicting the major steps of the program executedon pervasive computing device 10 in accordance with an embodiment of thepresent invention. The desired destination is entered at step 110. Thisis preferably accomplished using stylus 15 to enter an address ontographical display 12. The next step is determination of the currentlocation using the GPS system as shown in step 112.

From the current location, a map of the desired route is retrieved frommass storage device 54 or server 40 and displayed on graphical display12 as shown in step 114. The driver then embarks on the route as shownin step 116. The program then enters a continuous loop. The first stepof this loop is determining the current location as the vehicle travelsas shown in step 118. A determination is made of the distance from thiscurrent location to a next intersection on the route. If this distanceis greater than a selected distance stored with the intersection data,step 120 is negative and the program loops to step 118.

If the vehicle is within a given distance from the intersection, theimage of the intersection is fetched as shown in step 122. This imagemay be a perspective graphical image, a photograph or a video image.This image is displayed as shown in step 124. After the image isdisplayed, the program loops back to determine the new current locationat step 118. If the image is a moving video image, this new currentlocation is used to fetch the new perspective image relating to the newposition at step 124. In this way, the moving video image is kept insync with the viewpoint of the vehicle until the intersection has beenpassed.

Of course, the particular image displayed on the pervasive computingclient display need not be limited to a street or road. The image may beof any landmark, building, sign, road, or the like.

Although the various methods described are conveniently implemented in ageneral purpose computer selectively activated or reconfigured bysoftware, one of ordinary skill in the art would also recognize thatsuch methods may be carried out in hardware, in firmware, or in morespecialized apparatus constructed to perform the required method steps.

Further, as used herein, a pervasive computing “client” should bebroadly construed to mean any computer or component thereof directly orindirectly connected or connectable in any known or later-developedmanner to a computer network, such as the Internet. The term Web“server” should also be broadly construed to mean a computer, computerplatform, an adjunct to a computer or platform, or any componentthereof. Of course, a “client” should be broadly construed to mean onewho requests or gets the file, and “server” is the entity whichdownloads the file.

Having thus described our invention, what we claim as new and desire tosecure by Letters Patent is set forth in the following claims:
 1. Adisplay method operative in a pervasive computing client having agraphical display, comprising the steps of: displaying a map on thegraphical display; plotting a user's current position on the map; and asthe user's current position approaches a given physical landmark,displaying a photographic image of the landmark, the photographic imageshowing real-time conditions, to assist the user's navigation.
 2. Thedisplay method as described in claim 1 further including the step ofretrieving the photographic image prior to the step of displaying theimage.
 3. The display method as described in claim 2 wherein thephotographic image is retrieved from a storage media associated with thepervasive computing client.
 4. The display method as described in claim2 wherein the photographic image is retrieved from a server to which thepervasive computing client is connected.
 5. The display method asdescribed in claim 4 wherein the pervasive computing client is connectedto the server via a wireless connection.
 6. The display method asdescribed in claim 1 wherein the user's current position is derived froma global positioning system.
 7. The display method as described in claim1 wherein the photographic image is a frame of a video.
 8. The displaymethod as described in claim 1 further including the step of displayinga given graphical representation indicating a desired direction oftravel.
 9. The display method as described in claim 1 further includingthe step of outputting an audio cue indicating a desired direction oftravel.
 10. A display method operative in an in-vehicle pervasivecomputing client having a graphical display, comprising the steps of:displaying a map on the graphical display; plotting a vehicle's currentposition on the map; and as the vehicle's current position approaches agiven position relative to a physical landmark, selectively retrievingan image of the landmark, the image showing real-time conditions; anddisplaying the image.
 11. The display method as described in claim 10wherein the image is retrieved from a storage media associated with thepervasive computing client.
 12. The display method as described in claim10 wherein the photographic image is retrieved from a server to whichthe pervasive computing client is connected.
 13. The display method asdescribed in claim 12 wherein the pervasive computing client isconnected to the server via a wireless connection.
 14. The displaymethod as described in claim 10 wherein the vehicle's current positionis derived from a global positioning system.
 15. The display method asdescribed in claim 10 wherein the image is a photograph.
 16. The displaymethod as described in claim 15 wherein the photograph is collected inan off-line image collection process.
 17. The display method asdescribed in claim 10 further including the step of displaying a givengraphical representation indicating a desired direction of travel. 18.The display method as described in claim 10 further including the stepof outputting an audio cue indicating a desired direction of travel. 19.In a pervasive computing client device having a graphical display and abrowser, the improvement comprising: displaying a map on the graphicaldisplay; plotting a user's current position on the map; and as theuser's current position approaches a given physical landmark, displayinga photographic image of the landmark in the browser, the photographicimage showing real-time conditions.
 20. A pervasive computing client,comprising: a processor; a graphical display; means for displaying a mapon the graphical display; means for plotting a user's current positionon the map; and means responsive to the user's current positionapproaching a given physical landmark for displaying an image of thelandmark, the image showing real-time conditions.
 21. A computer programproduct in a computer-readable medium for use in a pervasive computingclient device having a graphical display, comprising: means fordisplaying a map on the graphical display; means for plotting a user'scurrent position on the map; and means responsive to the user's currentposition approaching a given physical landmark for retrieving anddisplaying an actual image of the landmark, the image showing real-timeconditions.
 22. The computer program product as described in claim 21further including means for displaying a graphical representation of anavigation aid in conjunction with the actual image.
 23. A method fordisplaying information in an in-vehicle pervasive computing clienthaving a browser, comprising the steps of: displaying a map representinga current location of the vehicle; as the user approaches a givenposition on the map, displaying an image of a physical landmark at thegiven position, the image showing real-time conditions.